Method and apparatus for monitoring individuals while protecting their privacy

ABSTRACT

A system for monitoring individuals while protecting their privacy includes at least one energy emitter configured to emit energy onto a field-of-view that may contain an individual, and at least one energy sensor configured to capture reflected energy from within the field-of-view. A spatial measurement module calculates spatial measurements of objects within the field-of-view based on data from the energy sensor. A schematic generation module creates schematic views of objects within the field-of-view, distinguishing human beings from each other and from inanimate objects or animals. Three-dimensional measurements of the individual and environs are transformed into a two-dimensional or other schematic view, allowing ongoing monitoring of the individual while preventing viewing of the individual&#39;s face and appearance, and preventing observation of what the individual may be wearing or watching.

PRIORITY CLAIM

This application claims the benefit of Provisional Patent ApplicationSer. No. 61/548,079, filed on Oct. 17, 2011, Provisional PatentApplication Ser. No. 61/561,627, filed on Nov. 18, 2011, ProvisionalPatent Application Ser. No. 61/567,940, filed on Dec. 7, 2011,Provisional Patent Application Ser. No. 61/663,889, filed on Jun. 25,2012, PCT International Application Serial No. PCT/US12/58443 filed onOct. 2, 2012, and PCT International Application Serial No.PCT/US12/58534 filed on Oct. 3, 2012. All of the above-identifiedprovisional patent applications and PCT international applications areincorporated by reference herein in their entirety.

BACKGROUND OF THE INVENTION

This invention relates to monitoring of an individual's presence andmovement. “Presence” as used herein refers to an individual's location,especially with reference to a particular room of a building (e.g., thepresence of a person within a specific location in a kitchen or a livingroom, such as standing in a corner or seated at a table). “Movement” asused herein refers to an individual's movement, especially withreference to leg, arm, or head movement (e.g., the movement of a personas the person walks from one point in a room to another, or the shiftingof a person's body or head posture while otherwise standing or sittingstill).

Monitoring an individual's presence and movement is useful in a widevariety of applications. Security cameras are commonplace in governmentand business facilities, and use thereof is increasing in privateresidences. GPS (global positioning satellite) sensors worn byindividuals suffering from dementia detect “wandering” behavior in thoseindividuals and alert caregivers. Baby monitors with video cameras allowparents to keep a vigilant eye on their children.

Monitoring for presence and movement comes with an important trade-off:the more precise and convenient the monitoring, the more invasive it isof an individuals' privacy. Security cameras—which are both precise andconvenient—record a person's appearance and visage, and using modernfacial recognition algorithms, can even reveal a person's identity. Suchcameras, which are precise and convenient, are highly invasive ofprivacy. On the other hand, worn GPS sensors are less invasive, but areless precise and less convenient.

Because of this trade-off, any type of monitoring that requiresprecision and convenience, coupled with privacy, cannot currently bereadily satisfied. An important example is the monitoring of olderindividuals in their own home. Specifically, it is desirable to monitorthe presence and movement of an older individual who lives alone, inorder to reassure friends and family that the individual is doing well,is able to move around, is not sick, has not fallen into a hypoglycemiccoma, has not fallen down, etc.

However, older individuals actively (and appropriately) are opposed toplacing video cameras or similar devices into their homes that will showwhat they are wearing, what they look like, what they are reading, whatthey are watching on TV, etc. There is an important unmet need tomonitor older individuals in their own homes while respecting andprotecting their privacy.

Known methods of monitoring presence and movement suffer from one ormore of the following disadvantages:

-   -   Known methods may generate visual data that show a person's face        or body, or show what the person is wearing, or show what the        person is reading or watching, thereby invading privacy;    -   Known methods may be susceptible to “reverse engineering”        whereby visual images of the individual may be reconstructed        from underlying data, even if the known methods attempt to        obfuscate or to hide such images;    -   Known methods may require that specific devices, such as GPS        devices, be physically worn by an individual;    -   Known methods may be imprecise and/or inconvenient;    -   Known methods may be unable to distinguish between multiple        persons at the same time, or between persons and animals.    -   Known methods may require a lighted environment to work, and        thus fail to work in the dark.

SUMMARY

To overcome the above-described problems, some embodiments of thepresent invention do not rely on video or image data streams. Instead,some embodiments of the inventive system and method rely on ongoingacquisition of “data-snapshots,” which are each obtained from a singleviewpoint, as described below, wherein each data-snapshot contains atleast depth data, and preferably a combination of depth and/or skeletondata and/or pixel label data.

Such data-snapshots may optionally be supplemented by one or more audiostreams in order to observe a person's voice, and/or one or more videostreams in order to observe a person's appearance—either over time, orat specific times (e.g., during an emergency). Measurements of anindividual's activity, gait, and posture—as well as a variety of otherbody measurements—may also be obtained and analyzed from one or moresuch data-snapshots, as also described in detail in the relatedapplications to which this application claims priority.

Embodiments of the present inventive system and method include thefollowing advantages, which are not intended as exhaustive list:

-   -   In some embodiments, it may utilize only one energy emitter and        only one camera, in order to obtain depth data, and/or skeleton        data, and/or pixel label data;    -   In some embodiments, the data used to monitor the individual is        of a nature such that the appearance of the individual cannot be        reconstructed from the data, thus ensuring that any visual        information about the individual—including, e.g., face, state of        dress, apparel being worn, what the individual is reading, what        the individual is watching—cannot be discovered or        “reverse-engineered” by others;    -   In some embodiments, it may utilize two cameras of        non-overlapping frequencies, in order to obtain at least depth        data, and preferably image data, skeleton data, and/or pixel        label data as well;    -   In some embodiments, it may be sized such that the overall        hardware components easily fit onto a shelf or at the base of a        television at home;    -   In some embodiments, it may monitor (and distinguish between)        multiple users in the same field-of-view at the same time;    -   It does not require any individuals being observed to wear any        sensors or devices or special clothing;    -   In some embodiments, it may operate in the dark (without the        need for exogenous light);    -   In some embodiments, it may operate on an ongoing basis, for        example, around-the-clock;    -   In some embodiments, it may not require the use of visual-light        camera or video, thus protecting the user's privacy so that, for        example, it is not possible to view the user's face or what the        viewer is wearing;    -   In some embodiments, it acquires detailed, high-resolution data        that are highly relevant to tracking presence and movement of        individuals, without revealing their face or appearance, or        without determining the activities in which they are engaged;    -   In some embodiments, it allows an optional visual or audio data        stream to be established, e.g., during emergencies;    -   It does not require a separate human operator (other than the        individual who is being monitored);    -   It is low-cost, compact, portable, and affordable;    -   It is easy to install and operate in an individual's own home;    -   It does not require special training or skills to operate;    -   It is able to supply health status measurements to third        parties, such as relatives, caregivers, or clinicians, in order        to enable reassurance and/or early warning and/or intervention;    -   It has low computational requirements, allowing it to execute        efficiently on low-end, affordable machinery;    -   It permits the user to move about while health status        measurements are being acquired, without having to pose or stay        still;    -   It is passive rather than active, and so does not require the        user to have to remember to perform particular actions or to        engage in particular activities;    -   It enables real-time interactivity with the user, for example,        it may adjust or respond in real-time to the user's movements,        or communicate useful information (such as health status        measurements displayed on a screen) to the user in real-time.

There are many useful applications of this low-cost, convenient methodand apparatus of monitoring presence and movement while protectingprivacy. The following recitation of useful applications is not intendedto be an exhaustive list, but merely points out the wide and divergentfields in which embodiments of the present inventive method findapplication.

Elderly individuals living alone at home are particularly vulnerable tosubtle health status deterioration that escapes detection until toolate. Such deterioration may be due to infection (e.g. pneumonia),malnutrition, depression, a recent surgery or hospital discharge, ormyriad other causes, and may initially appear minor: e.g., a somewhatmore-shuffling gait, a bit more time spent in bed each day, a softervolume of speech. But if not attended to, such deterioration mayprogress, steadily worsening until it reaches a “tipping point,” afterwhich the individual can no longer compensate, declines rapidly, andends up hospitalized, permanently institutionalized, or both.

Some embodiments of the present inventive method and apparatus allow asmall, affordable sensor to be placed within the individual's home thatallows friends, family, or other caregivers to monitor the individual'spresence and movement on a real-time basis, and to become aware if theindividual's presence or movement deviates from baseline or from typicaldaily routines, all the while ensuring that the individual's face, andappearance cannot be viewed.

Such a “silent guardian” offers benefits to caregivers and providers, inaddition to the individuals being guarded. Currently, caregivers mayendure round-the-clock concern and worry about their loved ones, such aselderly parents; embodiments of the present inventive method offerreassurance that caregivers will receive advance warning in manysituations that previously would have “slipped through the cracks.”Furthermore, providers, insurers, and the healthcare system overall, maybenefit from the lower costs stemming from prevented hospitalizations.

Embodiments of the present inventive method and apparatus may also beused to enhance care of a patient after discharge from a hospital (forexample, after a hip or knee surgery). For example, the presentinventive method and apparatus may be used by clinicians to ascertainthe frequency, intensity, and quality of ambulation of post-surgicalpatients, while preserving the patient's privacy.

Embodiments of the present inventive method and apparatus may also beplaced in care settings where patients are treated more intensively butare still ambulatory, such as hospital wards, assisted-livingfacilities, or nursing homes. In these cases, such embodiments canprovide early warning to on-site providers or care managers, thusreducing the need for round-the-clock human monitoring. The ability ofthe present inventive method and apparatus to simultaneously monitor,and distinguish between, multiple individuals is especially important inmulti-dweller or institutional settings.

The above examples show that embodiments of the present inventive methodand apparatus are useful in many applications across home and ambulatorycare, as well as in other applications such as security.

Specifically, one embodiment of the present inventive method includes:

-   -   a) Identifying and acquiring a collection of three-dimensional        depth measurements    -   b) Identify the presence and position of an individual and/or        other objects within the collection of three-dimensional depth        measurements    -   c) (Optionally) Transforming the depth measurements from a        three-dimensional collection of data into a two-dimensional        collection    -   d) Marking which elements of the three-dimensional and/or        two-dimensional collection correspond to the individual and/or        other objects being monitored    -   e) Transmitting the three-dimensional and/or two-dimensional        collection and/or marks to an operator and/or another user        and/or a storage system    -   f) (Optionally) Analyzing and/or reporting on the collection and        marks, including, e.g., text messages or emails    -   g) Repeating starting from step a) until all desired        measurements have been obtained

BRIEF DESCRIPTION OF THE DRAWINGS

FIGS. 1A and 1B show representations of depth data.

FIGS. 2A-2F show block diagrams according to specific embodiments of thepresent system and method.

FIGS. 3A and 3B show additional block diagrams according to specificembodiments of the present system and method.

FIG. 4 shows transformation of depth measurements from 3D to 2Daccording to a specific embodiment of the present inventive method.

FIG. 5 shows rapid transformation of 3D spatial measurements into 2Dschematics.

FIG. 6 shows a high-level flowchart according to a specific embodimentof the present inventive method.

FIG. 7 shows a screenshot example of a real-world embodiment of thepresent system and method.

FIG. 8 shows several actions that may be undertaken once a collection ofmeasurements (or schematic outputs) has been completed.

FIG. 9 shows an example of analytics, in this case a heat map, that maybe performed by embodiments of the present system and method.

FIG. 10 shows an example of a cartoon-like or schematized representationof objects in the field-of-view.

FIG. 11 shows an example of using legacy data when an inanimate objectin the field-of-view is temporarily blocked or hidden by another object.

DETAILED DESCRIPTION

Embodiments of the present invention are designed to enable presence andmovement monitoring of one or more individuals while protecting theprivacy of those individuals. The system may utilize a single energysensor to obtain, at a minimum, depth data; or two energy sensors ofnon-overlapping frequencies to obtain a combination of depth data andspectral data (for example, color image data). Skeleton data (whichconsists of the approximate locations in space of joints, or of otherambiguous and/or diffuse anatomic structures) may in turn be calculatedfrom the acquired depth and/or spectral data. Pixel label data (whichconsists of labeling pixels in acquired depth maps or color image maps,such that the labeled pixels correspond to the body surfaces of humansin the field-of-view) may also be calculated from the acquired depthand/or spectral data.

Any collection of distance measurements to (or between) objects in afield-of-view is referred to herein as “depth data”. There are many waysto acquire, calculate, or otherwise generate depth data for afield-of-view.

For example, depth data may be calculated based on a “time-of-flight”method. In this method, light with known physical characteristics (suchas wavelength) is emitted into a field-of-view. An energy sensor, suchas a camera, receives the light that is reflected from thefield-of-view. Changes in the physical characteristics of the lightbetween its being emitted and its being received—for example, theround-trip transit time of a light pulse, or the phase shift of anemitted waveform—allow calculation of the distance to various objects(that reflect the light) in the field-of-view.

If light pulses are utilized (for example, to measure round-trip transittime), the emitter can be, for example, a pulsed LED. If continuouslight is utilized (for example, to measure phase shift), the emitter canbe, for example, a laser. Time-of-flight cameras are a subset of LIDAR(Light Detection and Ranging) technologies, in whichemitted-and-reflected light is used to remotely gauge the distance orother properties of a target. LIDAR cameras are similar to radardevices; the main difference is that radar bounces radio waves offtarget objects, but LIDAR uses ultraviolet, visible, or near-infraredlight. Mesa Imaging AG, of Zurich, Switzerland, is an example of acompany that manufactures devices suitable to acquire depth data throughtime-of-flight: for example, its SR4000 time-of-flight camera.

Besides LIDAR, a different method of calculating depth data is throughthe use of “pattern deformation methods,” also sometimes called “lightcoding”. In pattern deformation methods, a light pattern with knownphysical characteristics (such as pattern shape and spacing) is emittedinto a field-of-view. An energy sensor, such as a camera, receives thelight pattern that is reflected from the field-of-view. Changes in thepattern between its being emitted and its being received—for example,gridlines moving closer further apart, or average distances betweenspeckled dots growing or shrinking—allow calculation of the distance tovarious objects (that reflect the light) in the field-of-view.

In contrast to time-of-flight or LIDAR, the specific wavelengths ortransit times of the emitted light are not crucial; what matters inpattern-deformation methods are the emitted pattern in which the lightis placed, and how that emitted pattern is subsequently reflected anddeformed by objects in the field-of-view. Because the specificwavelength is less important in pattern-deformation methods, a commonchoice of wavelength in such methods is infrared, which light cannot beseen by the human eye, and can be superimposed on a scene withoutdisturbing people. If the light pattern is relatively fixed andconstant, it is called “structured light”—often, structured-lightpatterns are grids of regular lines.

If the light pattern exhibits random or pseudorandom variation, it iscalled “coded light”—often, coded-light patterns are lattices of dots.The reason why random or pseudorandom variations may be used in lightpatterns is so that small areas of the pattern will “look slightlydifferent” compared to each other, enabling easier lining-up andregistration of the emitted and reflected patterns. PrimeSense Limited,of Tel Aviv, Israel, is an example of a company that manufacturessensors suitable to acquire depth data through pattern deformation. Itssensors are embedded in, for example, the Microsoft Kinect device(Microsoft Corp., Seattle, USA) and the Asus Xtion device (AsustekComputer Inc., Taipei, Taiwan).

Besides time-of-flight, LIDAR, and pattern deformation, a differentmethod of acquiring depth data is through the use of emitted energy thatis not light. For example, sound (rather than light) may be emitted andbounced off objects; the reflected physical characteristics of thesound, such as round-trip transit time, or frequency or phase shift, maybe used to calculate depth or other characteristics of the objects inthe field-of-view. Sommer Mess-Systemtechnik, of Koblach, Austria is anexample of a company that manufactures devices suitable to acquire depthdata through ultrasonic impulses: for example, its USH-8 sensor, whichuses ultrasonic impulses to measure snow depth.

Embodiments of the present invention may use any type of emitted andreceived energy, including but not limited to visible light, ultravioletlight, infrared light, radio waves, audible sound waves, ultrasonicfrequencies, and pressure vibrations, in order to acquire depth data.Embodiments of the present invention are agnostic as to the source ofdepth data. As used herein, “depth data” refers to measurements of thedistances to objects (or portions of objects) in a field-of-view.

Note that the term “camera” is used herein for convenience only, and anyenergy sensor, or image capture device, or energy capture device, ordata capture device using various ranges of electromagnetic radiation orother types of energy may be used and substituted therefore. The terms“energy sensor”, “camera,” “image capture device,” “energy capturedevice,” and “data capture device” are used interchangeably herein.

Some such devices need not emit electromagnetic radiation, because theycapture energy based on reflected radiation already present in theenvironment. Other such devices may emit electromagnetic radiation andcapture reflected radiation, such as ultrasonic transducers, and thelike, where such emitted electromagnetic or other energy radiation isnot present in the environment to a sufficient degree or sufficientlypresent in known directions relative to a target.

Additionally, the number of energy sensors are not limited to one or twosuch devices: one energy sensor, two energy sensors, or more than twoenergy sensors may be used (for example, to generate additionalstereoscopic data, or to cover a larger region of space), as well as asingle energy sensor.

“Image data” or “image” as used herein may refer to data or imagecaptured by any of the above-mentioned devices or sensors, such as anenergy sensor, a camera, an image capture device, an energy capturedevice, and/or a data capture device, and need not necessarily refer tothe optical range. In one embodiment, image data may refer to the samevisual-spectrum data that would be generated by a standard digitalcamera, consisting of a 2D photographic pixel map, where each pixelrepresents a visible color.

Note that in general, the term “color” as used herein may refer to allthe colors of the visual spectrum, or a grayscale spectrum, or any otherpalette of visual colors that are perceptible by the human eye. As usedherein, “color image data” refers to visual (visible to the human eye)image data, similar to that captured by a standard consumer digitalcamera.

“Depth data” is less intuitive than color image data. Depth datarepresents the distance from a sensor to a nearest object in space.FIGS. 1A and 1B show two representations of depth data. The preferredrepresentation of depth data, shown in FIG. 1A, is a 2D bitmap, alsosometimes referred to as a depth map. However, alternate representationsare also possible. The value of each (x, y) pixel in the 2D bitmap shownin FIG. 1A represents the distance from a common referenceplane—typically a vertical plane established by the sensor itself, withthe x-axis running horizontally, and the y-axis running vertically—tothe closest physical object, along a normal ray projected outward fromthe common reference plane at that (x, y) coordinate. In such acoordinate system, since the y-axis extends floor-to-ceiling, and thex-axis extends to left-and-right of the sensor, it follows that thez-axis extends straight out from the sensor into the field-of-view.

A 2D depth data bitmap therefore corresponds to a quantized contour, ortopographic, map of the sensor's field-of-view. Equivalently, a pixelvalue z at position (x, y) in the data bitmap indicates that the surface(or edge) of a real-world object exists at coordinate position (x, y, z)in physical space.

A depth bitmap can represent depth data only for aspects of an objectthat are visible to the sensor: any aspects of an object that areout-of-view of the viewpoint are “invisible” and not represented in thedepth bitmap.

For example, if we were to obtain a depth data bitmap of the Moon astaken from standing on the Earth, we would find that a collection ofpixels in the middle of the bitmap formed the shape of a circle. Thepixels in the center would have the lowest distance values (they wouldcorrespond to the central part of the Moon which is closest to theEarth), and the pixels at the edge of the circle would have the highestdistance values (they would correspond to the edge of the visible faceof the Moon). Pixels outside the circle of the Moon, representing thevoid of space, would have maximum distance values (essentiallyequivalent to infinity). The “dark side of the Moon”, invisible to us,would not be represented in the bitmap at all.

FIG. 1B shows an alternate representation of depth data, in which thepositions of objects in the field-of-view are described using a list ofangles and distances. Such a representation is not as advantageous asthe bitmap approach, due to the complexity of “working backwards” toidentify which objects are placed where in space.

FIG. 2A shows a block diagram of an embodiment of the present method andsystem. A system for monitoring individuals while protecting theirprivacy (the system) is shown generally as 200, which may be used tocarry out the method disclosed in this document. As set forth above, anyform of active energy capture (emission of energy and capture of thereflected energy) or passive energy capture (capture of reflected energybased on ambient energy sources) may be used.

As shown in FIG. 2A, energy emitter 202 bathes the field-of-view withenergy. As described previously, the energy emitted may include visiblelight, or non-visible light, or sound, or any other type of energy. Theenergy emitted may bathe the entire field-of-view all at once, or maybathe different parts of the field-in-view in turn. Energy sensor 204gathers the energy that is reflected or received from objects in thefield-of-view.

Depth calculation module 210 calculates the distances to objects in thefield-of-view using the information acquired by energy sensor 204. Asdescribed previously, such depth calculation may performed usingtime-of-flight, or LIDAR, or pattern deformation, or any other methodsuitable for calculating depth measurements. Depth calculation modulesupplies depth data 220, where, for example, depth data 220 may bestructured in a form similar to that shown in FIG. 1A.

In FIG. 2A, depth calculation module 210 uses the captured energy datafrom energy sensor 204 to calculate depth data 220 corresponding to theobjects in the field-of-view. Such calculation may also rely onknowledge of the characteristics of the most recent energycharacteristics or energy patterns emitted by energy emitter 202, and/oron past energy characteristics or energy patterns emitted by energyemitter 202, or captured by energy sensor 204, or on any otherinformation required to carry out depth calculations.

Sensor portion 201 encapsulates a minimal set of components required bysome embodiments of the present inventive method, viz., an energyemitter, an energy sensor, and a depth calculation module. Because ofthe similarity to energy sensor 204, optional color image sensor 206 isincluded for convenience within sensor portion 201. It is important tonote that sensor portion 201 is a label of convenience, roughlycorresponding to the typical hardware components required for somereal-world embodiments of the present inventive method, and so anycomponents of the present inventive method, including all of those, forexample, shown in FIG. 2A, may be brought in or out of sensor portion201. For example, optional skeleton calculation module 212 could appearinside sensor portion 201 in some embodiments of the present inventivemethod.

The depth data 220 may be used by optional skeleton calculation module212 in order to construct optional skeleton data 222, consisting of aset of approximate spatial locations of anatomic joints (e.g., the [x,y, z] locations of shoulder, hip, and ankle). The data from depthcalculation module 220 may also be used by optional pixel labelcalculation module 216 in order construct optional so-called “pixellabel” data 226, consisting of labeling individual pixels in a depth map(such as the depth map shown in FIG. 1A) that correspond to a humanbeing in the field-of-view. A wide variety of machine learning methodsare known in the art that may be utilized by optional skeletoncalculation module 212 and optional pixel label calculation module 216,and are not discussed further here.

Spatial measurement module 218 may use depth data 220 to calculatemeasurements in space, as described further below. Spatial measurementmodule 218 supplies spatial measurements 228. Spatial measurements 228may further include any of at least three categories of measurements:first, measurements pertaining to humans within the field-of-view (e.g.,the position of a person's arm, or where a person is standing within aroom); second, measurements of inanimate objects within thefield-of-view (e.g., the location of a table within a room); and third,measurements of animals within the field-of-view (e.g., where a dog iswalking within a room).

Schematic generation module 230 uses spatial measurements 228 to createschematic output 232. Schematic output 232 is a representation of thefield-of-view of sensor portion 201. For example, schematic output 232may be a representation of the field-of-view that does not use visualdata (photos or videos), in order to preserve the privacy of individualswithin the field-of-view. For example, schematic output 232 may be atwo-dimensional overhead representation of the field-of-view. Forexample, schematic output 232 may resemble a radar screen, in whichmoving objects (such as humans) are highlighted differently thanstationary objects (such as furniture). For example, schematic output232 may resemble a cartoon, in which cartoon-like avatars are used torepresent humans.

In some embodiments of the present invention, schematic generationmodule 230 may generate schematic output 232 in such a way as toidentify, recognize, or distinguish among different individuals. Thatis, if more than one person is in the field-of-view of sensor portion201, schematic output 232 may display information to individuallyhighlight or identify each person in the field-of-view.

For example, if two people are in the field-of-view, schematic output232 might display (for example) a green color when representing person#1, and a yellow color when representing person #2. For example, if twopeople are in the field-of-view, then schematic output 232 might use apredetermined icon or graphic representation for person #1, and adifferent predetermined icon or graphic representation for person #2.

There are multiple ways to accomplish recognition, identification, ordistinguishing among individuals, so that system 200 may distinguishamong two or more different individuals using a variety of methods. Insome embodiments of system 200, spatial measurements 228 may be used asbiometrics to distinguish individuals from each other. Examples ofbiometric measurements may include, for example, arm length,shoulder-to-shoulder width, and the person's height. In general,collections of spatial measurements of a person's body may be used, in amanner similar to a fingerprint, to identify that person, and/or todistinguish users of system 200 from each other. The biometric use ofspatial measurements 228 enjoys the advantage of not requiring colorimage data 224, thus helping to preserve privacy.

System 200 may further distinguish among two or more differentindividuals using facial recognition. Facial recognition methods arewell known in the art and not described further here. Color image data224 is, in general, required to enable facial recognition methods,because facial recognition methods usually require visual-spectruminformation. Some embodiments of the present invention utilize optionalcolor image data 224 in order to perform facial recognition and sorecognize or distinguish individuals in the field-of-view from eachother. In some embodiments of the present invention, so as to helppreserve privacy, optional color image data 224 is acquired temporarilyto enable facial recognition, but used for that purpose only briefly(e.g., in some embodiments, for approximately one second), and nottransmitted further to other systems or individuals.

In general, system 200 may measure or monitor more than one individualat substantially the same time. As described herein, whenever anymeasurements, schematic outputs, or any other data or processes are putforth that pertain to a single individual, they may also pertain to morethan one individual substantially simultaneously. For example, in someembodiments of system 200, spatial measurements 228 may be taken of morethan one individual substantially simultaneously. For example, in someembodiments of system 200, schematic output 232 may contain indicia,markings, identifications, or the like, that correspond to, ordistinguish among, more than one individual. Any utilization of thepresent invention for measuring or monitoring a single individual, maybe applied as well, and without loss of generality or capability, tomeasuring or monitoring multiple individuals.

Though not shown in FIG. 2A for brevity, schematic generation module 230may draw on any type of data in system 200—including depth data 220,optional skeleton data 222, optional pixel label data 226, or spatialmeasurements 228—to generate schematic output 232.

It is possible to calculate spatial measurements using only depth data220, and doing so helps to preserve privacy because it obviates the needto use color image data (or, more generally, any kind of photographic orvideo data) concerning the field-of-view. However, in some applicationsit may be preferable to also include a standard color image sensor 206,which gathers visual data in the same way as a standard digital camera.Optional color image sensor 206 supplies optional color image data 224.For example, if it desirable to open a video channel when an emergencyis suspected, then optional color image sensor 206 and optional colorimage data 224 can enable such a video channel. For many applications ofsystem 200, however, the color image sensor 206 and the color image data224 are optional.

As noted above, it is possible to calculate body measurements using onlydepth data 220. However, the speed of body measurement calculation maybe improved by drawing upon additional calculations performed on depthdata 220. For example, optional skeleton data 222 may be calculated fromdepth data 220, and used to improve the speed of calculating spatialmeasurements 228. For example, optional pixel label data 226 may becalculated from depth data 220, and used to improve the speed ofcalculating spatial measurements 228. As described previously, optionalskeleton data 222 describes the approximate spatial locations ofanatomic joints (for example, the three-dimensional [x, y, z] locationsof shoulder, hip, and ankle). As described previously, optional pixellabel data 226 distinguishes which pixels in a depth map (if any)correspond to a human being, and which do not.

In FIG. 2A, the depth data 220 consists of a set of calculated depthdata, where such data may conform, for example, to the representationshown in FIG. 1A. The optional color image data 224 consists of a set ofimage data; such data may, for example, be represented in the same wayas images that are acquired by a typical, everyday consumer digitalcamera, such as by using a pixel array or raster. The optional skeletondata 222 consists of a set of calculated spatial measurements of theapproximate locations of portions of a user's body, for example,shoulders and knees; such data may, for example, be represented by a setof (x,y,z) coordinates. The optional pixel label data 226 consists of aset of pixel labels delineating which pixels correspond to a human beingin the field-of-view; such data may, for example, be represented by apixel array or raster.

Embodiments of the system 200 may utilize a combination of depth data220, optional color image data 224, optional skeleton data 222, andoptional pixel label data 226, to conduct measurements of anindividual's body surface. The system 200 can utilize depth data 220alone, at the potential cost of decreased accuracy and/or speed in someembodiments.

The sensor portion 201 of FIG. 2A may alternately utilize more than twoimage sensors. For example, the sensor portion 201 of FIG. 2A may beaugmented with a third image sensor (not shown), which may overlap inenergy type or frequency with either the energy sensor 204 or theoptional color image sensor 206, in order to provide an additionalnearby stereoscopic vantage point by which to increase accuracy of depthcalculations. Or, multiple sensor portions 201 may be combined—forexample, by placing a different sensor portion 201 in each room of ahouse, then combining together their collective data to cover a largerarea than a single sensor portion 201 is capable of covering.

FIG. 2B shows another embodiment of the present inventive method. FIG.2B is similar to FIG. 2A, except that optional pixel label calculationmodule 216 and optional pixel label data 226 of FIG. 2A are omitted, toemphasize that they are not required for some embodiments. Items in FIG.2B correspond to their like-numbered items in FIG. 2A.

FIG. 2C shows another embodiment of the present inventive method. FIG.2C is similar to FIG. 2A, except that optional color image sensor 206and optional color image data 224 of FIG. 2A are omitted, to emphasizethat they are not required for some embodiments of the present inventivemethod. Items in FIG. 2C correspond to their like-numbered items in FIG.2A.

FIG. 2D shows another embodiment of the present inventive method. FIG.2D is similar to FIG. 2A, except that optional pixel label calculationmodule 216 and optional pixel label data 226 and optional color imagesensor 206 and optional color image data 224 of FIG. 2A are omitted, toemphasize that they are not required for some embodiments of the presentinventive method. Items in FIG. 2D correspond to their like-numbereditems in FIG. 2A.

FIG. 2E shows another embodiment of the present inventive method. FIG.2E is similar to FIG. 2A, except that optional skeleton calculationmodule 212 and optional skeleton data 222 and optional pixel labelcalculation module 216 and optional pixel label data 226 and optionalcolor image sensor 206 and optional color image data 224 of FIG. 2A areomitted, to emphasize that they are not required for some embodiments ofthe present inventive method. Items in FIG. 2E correspond to theirlike-numbered items in FIG. 2A.

FIG. 2F shows another embodiment 270 of the present inventive method.FIG. 2F shows an example of the present inventive method that usespattern-deformation and infrared (IR) light to acquire depthmeasurements. In FIG. 2F, IR pattern emitter 272 is analogous to energyemitter 202 of FIG. 2A. In FIG. 2F, IR pattern sensor 274 is analogousto energy sensor 204 of FIG. 2A. In FIG. 2F, optional color image sensor276 is analogous to optional color image sensor 206 of FIG. 2A. In FIG.2F, depth calculation module 280, optional skeleton calculation module282, depth data 290, optional skeleton data 292, and optional colorimage data 294, are analogous to their counterparts (respectively) 210,212, 220, 222, 224 of FIG. 2A.

In FIG. 2F, optional pattern pre-processing module 275 may clean,sharpen, remove noise from, or otherwise modify the information from IRpattern sensor 274. In FIG. 2F, optional color image pre-processingmodule 277 may clean, sharpen, remove noise from, or otherwise modifythe information from optional color image sensor 276.

Referring again to FIG. 2A, energy sensor 204 may optionally beaccompanied by a pre-processing module (not shown) analogous to optionalpattern pre-processing module 275. The optional color image sensor 206may optionally be accompanied by a pre-processing module (not shown)analogous to optional color image pre-processing module 277.Alternatively, in FIG. 2A, any pre-processing—if needed—analogous tocomponents 275 and 277 of FIG. 2F may be incorporated within(respectively) energy sensor 204 and optional color image sensor 206.

In FIG. 2F, depth calculation module 280 draws on the informationtransmitted by optional pattern pre-processing module 275—or directly onIR pattern sensor 274, if pattern pre-processing module 275 is notpresent—and may optionally also draw on the information transmitted byoptional color image pre-processing module 277—or optionally directly onoptional color image sensor 276, if color image pre-processing module277 is not present—in order to calculate depth data 290.

The color image itself, if present, may also be maintained separately asoptional color image data 294. The depth data calculation module 280does not require information from color image pre-processing module 277or optional color image sensor 276, but may optionally utilize suchinformation to improve the accuracy of depth data 290.

The data from any combination of IR pattern sensor 274, optional patternpre-processing module 275, optional color image sensor 276, optionalcolor image pre-processing module 277, and depth calculation module 280,may be used by optional skeleton calculation module 282 in order toconstruct optional skeleton data 292, consisting of a set of approximatespatial locations of anatomic joints (for example, the [x, y, z]locations of shoulder, hip, and ankle). Similar to the depth calculationmodule 280, the skeleton calculation module 282 requires onlyinformation from IR pattern sensor 274 and/or optional patternpre-processing module 275, and preferably information from depthcalculation module 280.

Although not shown in FIG. 2F, components analogous to optional pixellabel calculation module 216 and optional pixel label data 226 of FIG.2A may be placed in an analogous relationship in FIG. 2F as theircounterparts in FIG. 2A. For example, an optional pixel labelcalculation module in FIG. 2F (not shown) could receive the same inputsas optional skeleton calculation module 282, and produce optional pixellabel data (not shown), as described previously. For brevity, FIG. 2Fdoes not display such analogs to optional pixel label calculation module216 and optional pixel label data 226 of FIG. 2A.

Once the input data for body measurements (depth data 290, optionalskeleton data 292, optional color image data 294, and/or optional pixellabel data [not shown]) are obtained, the system 200 may utilize acomputer 298, including a processor 295, RAM 296, and ROM 297, toexecute a series of operations on the input data in order to producespatial measurements and to generate a schematic output, as describedfurther below. Alternatively, such processing may be performed bydedicated hardware chips and circuits, each of which may have their owninternal processor.

The resulting body surface measurements and schematic output may beplaced into a data storage device 284, shown on a display device 285,and/or transmitted over a communication interface 286, such as theInternet, or any suitable network. The system may be operated by theuser through user input 287; such input may include hand gestures, voicecommands, keyboard, mouse, joystick, game controller, or any other typeof user input.

In some embodiments of system 270, the depth calculation module 280 is acomponent of (or calculated by) computer 298, rather than sensor portion271. In some embodiments of system 270, the optional skeletoncalculation module 282 is a component of (or calculated by) computer298, rather than sensor portion 271. In some embodiments of system 270,the optional pixel label calculation module (not shown) is a componentof (or calculated by) computer 298, rather than sensor portion 271. Ingeneral, depth data 290, optional skeleton data 292, and optional pixellabel data (not shown) may be generated by modules at various pointswithin system 270, so that their generation is not limited to sensorportion 271.

Because system 200 and system 270 perform similar functions, and sharesimilar inputs and outputs, we will use “system 200” herein to referinterchangeably to both system 200 and system 270, unless otherwisenoted. Similarly, and for the same reasons, sensor portion 201 andsensor portion 271; energy emitter 202 and analogous IR light emitter272; energy sensor 204 and analogous IR pattern sensor 274; optionalcolor image sensor 206 and 276; depth calculation module 210 and 280;optional skeleton calculation module 212 and 282; depth data 220 and290; optional skeleton data 222 and 292; optional color image data 224and 294; will each be referred to interchangeably, unless otherwisenoted.

The system 200 (or system 270) may measure the user or environmentextremely quickly, and with minimal requirements to pose or position thebody. In particular, for an individual measurement of the user, thesystem 200 requires only a single data-snapshot of the user. Thus, insome embodiments, the user may need to stand relatively still for only apredetermined amount of time, for example 0.001 second to 0.1 second,which in an optical camera, may be determined by the amount of lighting,shutter speed, and aperture size. Other types of image capture or energycapture devices may operate on a much faster basis so that such captureis substantially instantaneous, at least from the perspective of theuser.

In other embodiments, the user need not necessarily stand in oneposition or maintain a particular position for any amount of time, andmay be able to move in real-time within the field of view of the imagecapture device. Individual measurements from different data-snapshotsmay also be combined or operated upon further, for example by addingthem or averaging them, as described below.

The term “data-snapshot” or “snapshot”, as used herein, refers to asingle set of depth, and/or image, and/or skeleton data, and/or pixellabel data, wherein the data are gathered substantially simultaneouslywith each other. As noted previously, a single data-snapshot cannotaccount for any “invisible” or “dark side” aspects of objects in thefield-of-view. Where necessary to complete a measurement, therefore, thesystem 200 may “fill in” for invisible aspects by using heuristics.

The original construction of optional skeleton data 222 may utilizemultiple calculations on depth and/or image data over time. The system200 is agnostic as to the means by which optional skeleton data 222 aregenerated. From the point of view of the system 200, asingle—substantially instantaneous—data-snapshot of depth, and/or image,and/or skeleton data, and/or pixel label data, is sufficient to obtain aparticular spatial measurement, regardless of the prior post-processingthat was necessary to generate the content of that data-snapshot.

Similarly, the original construction of depth data may utilize multiplecalculations on data received from either energy sensor 204 or optionalcolor image sensor 206 individually, or from both energy and color imagesensors 204 and 206 collectively over time. For example, a particularimage received at one moment in time by either energy sensor 204 oroptional color image sensor 206 may serve as a so-called reference imageat a subsequent moment in time, such that two or more images takenslightly apart in time are used to calculate depth data. Again, thesystem 200 is agnostic as to the means by which depth data, includingdepth data 220, are generated, including image processing that may occurover time, or different physical methods such as time-of-flight, LIDAR,or pattern deformation.

Through the use of a substantially instantaneous snapshot of data,gathered from one or more stationary cameras, the system 200 may avoidthe use of body-worn devices such as accelerometers, or the wearing ofspecial clothing, or the use of visual images such as from videocameras. As is described further below, this method also avoids the needfor manual intervention—in particular, the need for a second person toconduct body measurements. Some embodiments of the system 200 may bethought of as creating a “virtual radar” or a “smart radar” thatgenerates a privacy-respecting representation of a field-of-view. (Theuse of the term “radar” here is illustrative and analogous to thepopular depiction of radar-like screens in TV shows and movies; someembodiments of the system 200 may not use radar technologies per se.)

In some embodiments of system 200, energy sensor 204 and optional colorimage sensor 206 may be placed near each other, as a substantiallyco-located array, rather than being physically dispersed throughoutdifferent points on the perimeter of a field-of-view. Such co-locationis ideally as close as possible in order to have the field-of-view besimilar for each sensor. The feasible co-location separation distancedepends upon the size of the physical components. For example, if energysensor 204 and optional color image sensor 206 are instantiated as CMOSchips, the chips and their supporting electronics and optics may beplaced such that their borders are, for example, approximately 5 mmapart, and the centers of their lenses are, for example, approximately 2cm apart.

In general, the co-located sensors are preferably positioned with aseparation distance of millimeters to centimeters, although smaller andlarger distances are possible. Similarly, the angles of view of theco-located sensors are preferably within a few degrees of each other.This means that embodiments of the present system and method may be verycompact and portable, e.g., fitting easily on a shelf or at the base ofa television at home.

FIG. 3A shows another block diagram of an embodiment of the presentsystem and method. A system for monitoring individuals while protectingtheir privacy (the system) is shown generally as 300, which may be usedto carry out the method described in this document. As compared to FIG.2A, FIG. 3A demonstrates the addition of a sensor to detect audio(sound) in the environment, in particular, the sound of the user'sspeech. In FIG. 3, items 300 to 332, with the exception of 308 and 309,are equivalent to their like-numbered items in FIG. 2A. Audio sensor 308captures ambient audio, for example, through the use of a microphone ormicrophone array, and outputs audio data 309. Audio data 309 may berepresented in any way appropriate for conveying sound or an audiosignal, for example, as an analog waveform, or as a digital mp3 datafile.

FIG. 3B shows another block diagram of an embodiment of the presentsystem and method. A system for monitoring individuals while protectingtheir privacy (the system) is shown generally as 370, which may be usedto carry out the method described in this document. As compared to FIG.2F, FIG. 3B demonstrates the addition of a sensor to detect audio(sound) in the environment, in particular, the sound of the user'sspeech. In FIG. 3B, items 370 to 398, with the exception of 378, 379,and 396, are equivalent to their like-numbered items in FIG. 2F.

Audio sensor 378 captures ambient audio, for example, through the use ofa microphone or microphone array. Optional audio preprocessor 379carries out any desired preprocessing on the data received from audiosensor 378. An example of preprocessing would be identifying thephysical location in space, relative to the audio sensor 378, from whichthe sound emanated, by comparing two audio data streams (stereo signal)against each other. The output audio data 396 conveys any desiredcombination of raw and preprocessed audio data to other parts of thesystem 370, including the computer 398. Audio data 396 may berepresented in any suitable way appropriate for conveying sound or anaudio signal, for example, as an analog waveform, or as a digital mp3data file.

System 300 (or system 370), as a superset of system 200, can bydefinition perform all functions that system 200 (or system 270) canperform. For reasons of brevity, this document will often refer to“system 200” instead of “system 200 and/or system 270 and/or system 300and/or system 370”, but it should be understood that “system 300” (or“system 370”) can be substituted in place of “system 200” (or “system270”). The converse is not true, because system 300 (or system 370)possesses audio capabilities that system 200 (or system 270) does notpossess.

Depth calculation module 210, optional skeleton calculation module 212,optional pixel label calculation module 216, spatial measurement module218, schematic generation module 230, and all other modules describedherein, may be implemented in circuitry as a physical component orprocessing element, whether integrated or discrete, or may beimplemented to the extent possible, in software to be executed by theprocessor or specialized processing circuitry.

For a single measurement, certain embodiments of the system 200 mayrequire only a single data-snapshot of the user, taken from a singlepoint of view. This is because the system 200 may use heuristics—such asthe inherent symmetry of the human body—to “fill in”, or compensate for,any invisible depth or image information that is invisible to the sensorportion 201. Furthermore, multiple measurements may be drawn from asingle snapshot.

One reason that multiple data snapshots may be preferable is due tonoise in the system. If the inputs or outputs at any component of sensorportion 201 are noisy—that is, varying randomly or non-randomly, dueeither to inherent aspects of sensor portion 201 or to externalenvironmental conditions, then multiple data snapshots may be requiredto extract improved signal from the noisy background. For example, datasnapshots may be averaged over time, using signal processing methods, inorder to have noise “cancel out” and thereby diminish over time, whileconstructively adding together (strengthening) the valuable signal. Ifsuch averaging over time is performed, then multiple data snapshots maybe required for higher-accuracy measurements.

One reason that multiple data snapshots may be required is to track orto monitor spatial measurements over time. For example, tracking thelocation of an individual within a room over time requires correspondingmultiple data snapshots over time. In some embodiments of the system200, to track measurements that change over time, measurements may beacquired at a sampling rate ranging from approximately 30 data snapshotsper second, to approximately 1 data snapshot per 30 seconds. Theduration of time during which measurements are tracked may bepredetermined. For example, in some embodiments of the system 200,measurements may be carried out on an ongoing basis, indefinitely (e.g.,until the user chooses to stop the system 200 from running). In otherembodiments of the system 200, measurements may be carried out onlyduring certain time intervals (e.g., only during daytime).

Therefore, although any one measurement may require only a singlesnapshot, nonetheless, in some embodiments, more than one snapshot maybe used to obtain a complete set of desired measurements, or to trackhow measurements change over time.

FIGS. 4A and 4B show an example of transforming the three-dimensionaldepth measurements acquired by system 200 into a two-dimensionalrepresentation that is suitable for monitoring an individual whilepreserving privacy. FIG. 4A shows an example of an individual 410standing in a living room 400. The sensor portion 201 is embodied as asensor device 401 placed, in this example, on a table or counter withinliving room 400. The living room 400 may contain a variety of furniture,for example, television set 420 and couch 415. In the example of FIG. 4,the system 200 may utilize sensor portion 201 (embodied as sensor device401) to acquire depth data 220 from a field-of-view within the livingroom 400.

Each element of the depth data—that is, each pixel in the depth map ofFIG. 1A—corresponds, as shown in FIG. 1A, to a distance from the sensordevice 401 to a nearest-object in the field-of-view. Furthermore, in theexample of FIG. 4, the system 200 may use the depth data to identify thepresence of human being 410 in the field-of-view. For example, optionalpixel label data 226 may be used to identify the presence of human being410. Note that system 200 may also identify the presence of multiplehuman beings in the field-of-view, and distinguish them from each other;this is not shown in FIG. 4.

FIG. 4B shows the result of mathematically transforming the 3D depthdata of the living room 400 into a 2D depth data representation 450. The2D data representation 450, in the example of FIG. 4, is an overheadschematic of the living room 400. Note that any schematic, such as aside schematic or even angular-slice schematic, could have beengenerated instead of an overhead schematic. The 2D pixels representingthe individual being monitored 460 correspond to the 3D depthmeasurements that were identified by the system 200 as belonging to thehuman being 410, as described further below. The 2D data representationof the television set 470 corresponds to the living room television set420, and the 2D representation of the couch 465 corresponds to theliving room couch 415.

The 2D depth representation 450 can be created from the 3D depth data ofthe living room 400 in a variety of ways. One way is to create 3Dcomputational “meshes” from the 3D data, as is known in the art, andthen to mathematically shift the user's viewpoint of that mesh to anoverhead view.

FIGS. 5A-5C shows a simpler and more rapid way of transforming the 3Ddepth data to a 2D schematic view. The method shown in FIGS. 5A-5Cenjoys the advantage of low computational requirements, allowing it torun on low-end, affordable hardware embodiments. FIG. 5A shows a personstanding in a field-of-view, flanked on either side by an inanimateobject. The nearest border of the person is 80 cm from (the verticalplane defined by) sensor portion 201; the nearest border of object 1 is100 cm from (the vertical plane defined by) sensor portion 201; andseveral nearest points along the border of object 2 are 100, 110, and120 cm from (the vertical plane defined by) sensor portion 201.

FIG. 5B shows a depth map produced by sensor portion 201. This depth mapcorresponds to depth data 220 in FIG. 2A. The diagonally-shaded pixelscorrespond to the nearest border of the person at 80 cm, atapproximately the middle of the field-of-view. The black-shaded andgrey-shaded pixels in FIG. 5B similarly correspond to borders of objects1 and 2, flanking the person.

In FIG. 5B, the depth map pixels corresponding to a person are shadedwith a diagonal pattern to highlight the fact that, for example, ifoptional pixel label data 226 are available, the pixels corresponding tothe person can be “labeled” as belonging to a person and not to aninanimate object.

FIG. 5C shows a two-dimensional (2D) schematic 550 of the field-of-view,generated from the depth map in FIG. 5B. Wherever a pixel appears in thedepth map of FIG. 5B, a corresponding pixel is placed in the 2Dschematic 550, at the appropriate depth-value distance for that pixel.Multiple pixels in the depth map may correspond to the same pixel in theschematic. For example, each of pixels 520, 521, and 522 in the depthmap of FIG. 5B are transformed into the same place in FIG. 5C, pixel540, corresponding to a distance of 100 cm from sensor portion 201. Ifpixel 520 had contained a greater depth value than pixels 521—forexample, if object 1 were shaped like a triangle vertically, so that itstop part was farther away from sensor portion 201 than its bottompart—then pixel 520 would have been transformed into a different placein FIG. 5C, shown as pixel 541.

As shown in FIG. 5C, schematic pixels may be colored or labeled toconvey additional information. For example, schematic pixels may becolored blue if they correspond to an inanimate object, and coloredgreen if they correspond to a human being.

In some embodiments of system 200, 2D schematic 550 may show only theoutlines or “shells” of the various objects in the field-of-view. Thismay happen when sensor portion 201 is unable to generate depth datacorresponding to the “dark sides” or hidden aspects of objects. Forexample, the 2D schematic 550 contains pixels only corresponding to thesensor-facing aspects of the person and the two objects in FIG. 5A. Theback of the person, and the backs of object 1 and object 2, are absentfrom 2D schematic 550.

In some embodiments of system 200, 2D schematic 550 may be augmentedwith “memory” about objects in the field-of-view. As used herein,“legacy data” refers to information stored about objects in thefield-of-view. For example, depth data corresponding to objects in thefield of view may be stored as legacy data. For example, if data aboutobject 1 (in FIG. 5A) was acquired by system 200; and subsequently, aperson entered the field-of-view and stood between sensor portion 201and object 1, thus hiding or “shadowing” object 1 from view; then system200 could still “fill in” object 1 in 2D schematic 550 using past legacydata about object 1.

Returning to FIGS. 4A and 4B, the transformation from 3D depth data into2D representation can be performed in a wide variety of ways and resultin a wide variety of representations. For example, the overhead view450, corresponding to schematic output 232, could have been rendered asa side view (as opposed to an overhead view) of the individual 410. Forexample, schematic output 232 could be represented as an animatedstick-figure or cartoon of a user, mirroring the user's movement overtime.

As demonstrated in the example of FIG. 4, some embodiments of thepresent invention are evocative of aircraft radar or medical ultrasound.This is because such embodiments of the present invention acquirenon-visual data—specifically, depth data—and use that depth data toreconstruct a view of a scene that cannot be used to discover the visualappearance of objects in the scene, which is analogous to the wayaircraft radar or medical ultrasound reconstructs a view of a scene thatcannot be used to discover the visual appearance of objects in thescene.

However, the mechanisms through which such views are constructed arevery different that the embodiments of the present invention, and bothaircraft radar and medical ultrasound, and their purposes andmechanisms, are completely different. The analogy herein is intendedonly to aid in comprehension of the present invention's utility. In theexample of FIG. 4, clearly aircraft radar and medical ultrasound wouldnot be feasible technologies for monitoring an individual in a livingroom.

Returning to FIG. 5, embodiments of system 200 may use optional skeletondata 222 and/or optional pixel label data 226 to identify which pixelscorrespond to individuals, and which pixels correspond to inanimateobjects.

Skeleton data 222, if present, generally consist of approximatelocations of nebulously defined portions of the body, or collections ofanatomic structures. Skeleton data can be thought of as guideposts orlandmarks of general regions of the human body. Most often, theycorrespond to joints of the human skeleton, such as the shoulder orknee, because machine recognition algorithms may be employed torecognize structures that stay relatively constant in shape whilemoving, such as arms and legs, and therefore these algorithms may alsobe used to identify the approximate articulation regions between, say,arms and legs.

An example of skeleton data would be the approximate 3D spatial locationof the right shoulder joint. The right shoulder joint is of nebulousdefinition both structurally and spatially; it consists of multipleanatomic components (portions of the arm, ribcage, surroundingmusculature, and so forth) and cannot be precisely located on the humanbody, only approximately outlined.

Returning to FIG. 5, skeleton data 222 may be used to help label pixelsthat are located near specific anatomic regions of an individual's body.Pixel label data 226, if present, consist of labels that may be appliedto individual pixels in depth data 220, or to individual pixels inoptional color image data 224. The use of these labels, when they arepresent, is to distinguish human beings from each other, and from theambient environment, in a field-of-view.

For example, if depth data 220 were represented by a 640 by 480 pixeldepth map of a field-of-view, and if the depth pixel at coordinate (400,200) corresponded to a distance to a portion of the body surface of ahuman being; the depth pixel at coordinate (500, 300) corresponded to adistance to a portion of the body surface of a different human being;and the depth pixel at coordinate (20, 50) corresponded to a distance toa door or a wall in the local environment, then depth pixel (400, 200)might be labeled “person #1”, depth pixel (500, 300) might be labeled“person #2”, and depth pixel (20, 50) might be labeled “non-person”.

Similar reasoning applies to optional color image data 224. In sum, ifdepth data 220 or optional color image data 224 are represented aspixels—for example, in an array or raster representation—such pixels maybe attached with labels that distinguish whether the pixel correspondsto a person or a non-person, and if a person, an arbitrary identifierfor the person, where such labels are maintained in system 200 asoptional pixel label data 226.

Both optional skeleton data 222 and optional pixel label data 226generally cannot be used to precisely locate and track (over time) aspecific portion of the human body. Optional pixel label data 226 aregenerally able to signify, for example, that a specific pixel in aparticular data snapshot belongs to a surface of a human body and notthe ambient environment; or that two different pixels belong to twodifferent human bodies.

Optional pixel label data 266 generally cannot uniquely identify aperson's identity (for example, they cannot label that a person is “JohnH. Watson who lives at 221B Baker Street”, as opposed to “person #1”),nor can optional pixel label data 226 generally label a portion of abody (for example, they cannot label that a pixel belongs to “person#1's right shoulder” as opposed to just “person #1”). Optional pixellabel data 266 are therefore equivalent to a type of “mask”, as the termis known in computer science—applying this pixel label “mask” to depthdata 220 or to optional color image data 224 highlights which pixels, ifany, correspond to an arbitrarily numbered human being.

Returning to FIG. 5, pixel label data 226 may be used to help labelpixels that correspond to one or more individuals in the field-of-view.

A wide variety of methods to calculate skeleton data and/or pixel labeldata as outputs, using depth data and/or color image data as inputs, areknown in the art, and may draw upon machine learning, statistical, orother technologies or methods. For example, the Microsoft Kinect ForWindows Software Development Kit (SDK), from Microsoft Corp. of Seattle,USA, provides software routines to calculate skeleton data and pixellabel data (called “player identification” in the Kinect for WindowsSDK) from depth data and/or color image data.

For example, the OpenNI open-source software framework, under theauspices of the OpenNI Organization, similarly provides softwareroutines to calculate skeleton data (called “joint data” in OpenNI) andpixel label data (called “figure identification” in OpenNI) from depthdata and/or color image data. The Kinect for Windows SDK and the OpenNIframework employ different computational methods, utilize differentAPIs, have different operating characteristics, and representinformation differently. They are mentioned here as illustrations ofpotential methods, which are commercially available, to calculateskeleton data 222 or pixel label data 226. The system 200 is agnostic asto the means by which skeleton data 222 or pixel label data 226 aregenerated.

Note that some embodiments of the present inventive method use types ofenergy, such as infrared light from IR light emitter 272, that cannotpenetrate worn garments. Other embodiments may employ energy patternsthat are able to penetrate worn garments. However, because suchpenetrating radiation may be harmful to human health, or may poseprivacy hazards, some embodiments preferably rely on emitted energy oftypes, such as infrared, that do not penetrate worn garments.

For many applications, such as gait analysis, it is important to be ableto measure either the surface of the human body directly, or ofinterposed worn garments that closely approximate the surface of thehuman body. As a result, some embodiments may place constraints on thenature of the clothing worn during execution of a particularapplication. For example, an application to track smoothness of armmotion for a Parkinson's Disease patient may require the user to wear arelatively tight-fitting shirt, rather than, say, a billowy parka.

In the descriptions and Figures that follow, it should be appreciatedthat only depth data are required to carry out body measurements. Forexample, depth data alone—or, optionally, a combination of depth data,and the skeleton data that are calculated from the depth data—may besufficient to carry out a measurement of stride length, because suchdata may enable identification of all necessary body landmarks (e.g.,points on the foot and ankle) and measure distances between thoselandmarks.

In other cases, depth data are preferably combined with color imagedata, or a combination of depth data, calculated skeleton data,calculated pixel label data, and color image data may be preferable. Ingeneral, identifying the position of a body landmark requires utilizingsome combination of depth data 220, optional skeleton data 222, optionalpixel label data 226, and optional color image data 224, but thespecific combination, and the specific requisite calculations carriedout on that combination, differ from landmark to landmark.

FIG. 6 shows a high-level flowchart describing a preferred embodiment ofthe present inventive method, beginning at step 600. In Step 605, acollection of 3D measurements is identified and acquired. For example,Step 605 might include the collection of measurements relevant to anindividual standing, walking, or sitting in his/her living room. In Step610, the presence and position of the individual(s) being monitored isidentified within the collection of 3D depth measurements—in otherwords, the individual is segmented out of the 3D depth data. This may bedone through a variety of methods. For example, presence and position ofthe individual in step 610 may be ascertained in detail through bodymeasurements, as described above, and also described in patentapplications referenced in the above priority claim, which areincorporated herein by reference.

For example, presence and position of the individual in step 610 may beascertained approximately by performing a so-called “diff” or“difference” operation over time, wherein those depth pixels that changevalues substantially are assumed to correspond to a moving human being,and those depth values that stay substantially constant are assumed tocorrespond to inanimate objects such as furniture.

Note that step 610, in identifying an individual within thefield-of-view, also indirectly identifies inanimate objects, such asfurniture. For example, in some embodiments of system 200, any objectthat is not a human may be assumed to be an inanimate object. Forexample, in some embodiments of system 200, any object that is not ahuman, but which location changes over a time period of approximatelyseconds to minutes, may be assumed to be an animal. Though not shown inFIG. 6, some embodiments of system 200 may store measurements ofinanimate objects over time as legacy data, as described previously, sothat if, for example, a human being enters the field-of-view andtemporarily blocks the view by sensor portion 201 of the inanimateobjects, the legacy data of the inanimate objects may be used to “fillin” and restore the temporarily-blocked depth or other related data.

This is useful, for example, to fill in “shadows” in the depth datacaused by a human being standing between sensor 201 and an object suchas a chair or table—in other words, since it is unlikely that the chairhas moved on its own, system 200 can continue to draw or represent thechair using its legacy data of the field-of-view.

In Step 615, the 3D depth data are optionally rendered into a 2Dview—for example, an overhead view, or a side view. FIG. 5 shows oneexample of how this rendering or transformation may be accomplished insome embodiments of system 200; in addition, many other methods ofconverting 3D data to 2D data are known in the art. In step 615, the 2Dview may be further modified or made representational. For example, instep 615, elements of the 2D view may be blurred or otherwise obscured(for example, to enhance privacy). For example, in step 615, elements ofthe 2D view may be turned into indicia or graphics, such as cartoon-likerepresentations.

In step 620, the elements of either the 3D or 2D data collection thatcorrespond to the individual being monitored are marked. FIG. 5 shows anexample of how each of these steps may be accomplished in someembodiments of system 200. As described in the previous paragraph, insteps 615 and 620, system 200 may use previously-collected informationas legacy data (not shown in FIG. 6) of inanimate objects to fill inshapes or pixels in cases where the user is standing between sensorportion 201 and an inanimate object. Further, in step 620, elementscorresponding to humans, animals, or inanimate objects may be turnedinto indicia or graphics. For example, in step 620, objects in thefield-of-view corresponding to humans, animals, or furniture, may berepresented as schematized or cartoon-like representations of(respectively) humans, animals, or furniture. An example of suchrepresentation is shown in FIG. 10.

Some embodiments of system 200 may further distinguish between humans,animals, and inanimate objects. For example, schematic output 232 mayrecognize, identify, label, or otherwise distinguish humans, animals,and inanimate objects from each other. There are many ways for system200 to recognize whether an object in the field-of-view is a human,animal, or inanimate object.

For example, in some embodiments of system 200, if optional skeletondata 222 or optional pixel label data 226 are available, they may beused (singly or in combination) to identify those objects in thefield-of-view that are humans. For example, the machine learningalgorithms that generate optional skeleton data 222 or optional pixellabel data 226 may recognize the presence of a moving sphere-shapedobject (head) with two moving cylinder-shaped objects below it (arms),which combination distinguishes a human from either an animal or aninanimate object.

In some embodiments of system 200, if only depth data 220 (not optionalskeleton data 222 or optional pixel label data 226 or optional colorimage data 224) is available, then many methods are still available todistinguish humans, animals, and inanimate objects from each other. Forexample, if a collection of depth measurements does not varysubstantially (for example, beyond the threshold of statistical noise)over a period of days to weeks, then that collection of depthmeasurements likely corresponds to an inanimate object.

For example, if a collection of depth measurements moves over a shortperiod of time (for example, within a few seconds or minutes), but thedepth measurements are all of an object relatively low in height (forexample, less than a half-meter above the floor), than that collectionof depth measurements likely corresponds to a pet. Conversely, depthmeasurements of a relatively tall moving object (for example, more thana meter tall) are likely to correspond to a human. The methods describedin this paragraph are examples that serve to illustrate the wide varietyof methods that may be used to recognize and distinguish among humans,animals, and inanimate objects. In general, a wide variety of methodsare described in the art, ranging from (for example) statistical toneural network to rules-based, that may be used by system 200 torecognize and distinguish among humans, animals, and inanimate objects.

In Step 625, the collection of 3D and/or 2D data are transmitted to anoperator or other user, and/or transmitted to other devices, such as astorage system. Note that step 625 may transmit information to othersusers of similar systems 200. For example, in some embodiments of system200, multiple users are able to “watch over” each other.

In Step 630, the collection of 3D and/or 2D data are optionallyanalyzed, either by human or automated algorithms, and text messages,alerts, warnings, and the like are optionally sent to mobile phones,computers, or other devices. For example, Step 630 might send an alertas a text message to a designated mobile phone, or as a security alertto authorized personnel.

Step 630 further allows a wide variety of other operations,calculations, or derivative measurements to be performed on measurementsof the individual being monitored—for example, gait analysis, orclothing size measurement, as described in detail in relatedapplications mentioned above. For example, step 630 may first calculatethe centroid of 2D depth measurements corresponding to an individualwithin the field-of-view, and then further calculate the walking speedof the individual by comparing sequential such centroid calculationsover time.

Step 640 evaluates the results of Steps 605-630 to decide whether alldesired measurements have been obtained. Typically, it is preferable tocontinuously loop through the steps shown the flowchart of FIG. 6 aslong as monitoring of the individual is active. Step 650 completes theflowchart.

FIG. 7 shows a screenshot example of a real-world embodiment of thepresent inventive method. The 2D schematic 700 is similar to 2D depthrepresentation 450 of FIG. 4B, or to the 2D schematic 550 of FIG. 5C.The 2D schematic 700 is the output of step 620 in FIG. 6, andcorresponds to schematic output 232 of FIG. 2. In FIG. 7, a user 710(which may be, for example, highlighted in green) is walking through thefield-of-view of sensor portion 201. Around the user are various wallsand furniture 720 (which may be, for example, highlighted in blue).

A list of the names of multiple users 730 appears in FIG. 7. In theexample of FIG. 7, each of these users is in a different physicallocation—for example, the user's own home—and each user is supplied witha separate instance of system 200. These users are able to select eachother's names, and by so doing, to switch the view of system 200 ontoeach other. This is an example of how system 200 enables theseindividuals to “watch over” each other, while preserving their privacy(because no photos or videos need to be acquired or transmitted). Beingable to switch the 2D schematic 700 in order to view different users 730is an example of step 625 of FIG. 6.

Alert listing 740 in FIG. 7 exhibits a series of alerts pertaining toone of the users, Anne. System 200 detects, at various times, that Annehas appeared in the field-of-view; that Anne has remained motionless fora period of time in the field-of-view; and that Anne has left the fieldof view. In FIG. 7, these alerts appear in alert listing 740, and aretransmitted to others users as well (when a view is selected from thelist of users 730). Alert listing 740 is an example of an output of Step630 in FIG. 6. The alerts shown in FIG. 7 are examples, and notexhaustive of the alerts that may be generated by system 200.

FIG. 8, which begins at Step 800, demonstrates the range of actions thatmay be performed once a collection of measurements, as described in FIG.6 (Step 605), is complete. For example, FIG. 6, Step 630, may invoke thesteps of FIG. 8 one or more times. The actions that may be taken oncemeasurements are gathered, as shown in FIG. 8, include but are notlimited to:

Step 805: store, adjust, or transmit the measurements, schematicoutputs, or other parameters. For example, transmission of measurementsor schematic outputs may occur via the internet, to a friend or aclinical facility that can monitor the user for signs of health statusdecline; or locally, to a disk storage, so as to retain and chartmeasurements over time. Measurements, schematic outputs, or otherparameters may also be adjusted, for example, to match requirements fordata structure or for data compression or for clinical use before beingtransmitted to another system or party. The term “parameter” hereinrefers to any aspect of the user, such as demographic data, orlaboratory values from third-party devices (such as glucometers or bloodpressure cuffs). (Note that in some embodiments, color image data 224,depth data 220, and optional skeleton data 222 are preferably notretained nor stored by the system 200, in order to preserve the privacyof the user.)

Step 810: combine the measurements to generate new measurements, e.g.,the average walking speed of the user during a period of one day may becalculated by averaging several walking speed measurements taken atvarious times during that day.

Step 815: compare different measurements to improve the accuracy of themeasuring process. For example, objects in the field-of-view whosemeasurements do not change over a period of several weeks may beassigned a high confidence that those measurements correspond tofurniture.

Step 815 may also perform an additional calibration check on the systemas a whole, by taking measurements of known objects using differentdata-snapshots, and then comparing the measurements to check forconsistency.

Step 820: compare or contrast measurements over time. This allowsmeasurements to be charted or trended over time. For example, a “heatmap” of the field-of-view may be generated, in which darker-coloredareas highlight where the user has spent more time walking or sitting,and lighter-colored areas highlight where the user has rarely or neverbeen present. FIG. 9 shows an example of a heat map.

Step 825: compare measurements or schematic outputs against userbaselines or other types of benchmarks, or to other comparators (forexample, measurements or schematic outputs generated for other users ofsystem 200). Step 825 may perform comparisons using thresholds,statistical methods, or any other methods that enable a comparison ofmeasurements, or any other data, over space or time.

The routine exits at Step 850.

The actions listed in FIG. 8 may be combined in any number, order, orsequence, and are further not intended to be exhaustive. The scope ofthe system and method includes any actions that may be performed uponthe gathered measurements or schematic outputs.

FIG. 9 shows a “heat map” of an individual's location within a room,displaying widely-spaced representations of the individual where he/shespends less time, and closer-spaced representations where he/she spendsmore time. Such a heat map demonstrates, for example, that over time, anindividual was spending increasingly more time sitting on a couch, andless time actively walking around. (Baseline and deviation measurementsare not explicitly shown in FIG. 9.) The heat map of FIG. 9 may begenerated, for example, by FIG. 8 Step 820.

FIG. 10 shows an example of representing objects within thefield-of-view as schematized or cartoon-like figures in schematic output232. FIG. 10A shows a schematic output 1000, corresponding to thefield-of-view of sensor portion 1005, within which are located a human1010, a dog 1015, a couch 1020, and a TV 1030. FIG. 10B shows acorresponding schematic output 1050, in which the representation of thehuman 1010 has been converted to a cartoon-like FIG. 1060, and therepresentation of the dog 1015 has been converted to a cartoon-like FIG.1065. The cartoon-like figures of FIG. 10B may be generated, forexample, by FIG. 6 Step 615 or Step 620.

The cartoon-like figures of FIG. 10B (and, in general, anyrepresentational indicia or graphics utilized by system 200) may bevaried by system 200, singly or over time. For example, in schematicoutput 1050, an icon of a seated stick figure might be used to representan individual whenever that individual is sitting down, but a differenticon of a standing stick figure might be used to represent the sameindividual whenever that individual is upright. For example, inschematic output 1050, an icon of an eating animal might be used torepresent a dog whenever that dog is near the known location of itswater bowl, and an icon of a non-eating animal might be used torepresent the same dog otherwise.

FIG. 11 shows an example of using legacy data to fill in missinginformation about objects in the field-of-view that are temporarilyhidden from view of sensor portion 201. FIG. 11A shows a schematicoutput 1100, corresponding to the field-of-view of sensor portion 1105,within which are located a human 1110 and a couch 1120. In the exampleof FIG. 11A, system 200 recognizes that couch 1120 is an inanimateobject (for example, because the couch remains stationary for days orweeks), and so system 200 retains depth measurements of couch 1120 aslegacy data. FIG. 11B shows a schematic output 1130 in which legacy datais not used. In FIG. 11B, the human 1130 has changed location within theroom, and is now temporarily blocking the sensor's view of couch 1120.As a result, some of the depth measurements of couch 1120 can no longerbe obtained by sensor portion 1105, and so, without the use of legacydata, the couch appears in schematic output 1130 as having a “hole” or a“shadow” 1150 that is caused by the presence of human 1140. FIG. 11Cshows a schematic output 1160 in which legacy data is used. In FIG. 11C,the system 200 “fills in” the missing information for couch 1180 usinglegacy data, so that couch 1180 appears substantially similar to couch1120, even though sensor portion 1105 remains temporarily blocked fromview by human 1170. Although this example shows an example of legacydata for a single couch, it is apparent that storage and use of legacydata may be applied to any or all objects in a field-of-view, or toportions of those objects; may correspond to one or to multiple timeperiods; and may be utilized over any desired duration (or durations) oftime.

As mentioned earlier, embodiments of the present inventive method may beused in a wide variety of applications. For example, in monitoringindividuals while protecting their privacy, embodiments of the presentinventive method may be employed to measure entrance into afield-of-view; exit from a field-of-view; duration of time spent moving,or spent motionless, within a field-of-view; duration of time spentsitting, or standing, within a field-of-view; noting whether a userreaches for, or points to, an object; and many other types ofmeasurements. The schematics generated by system 200 may be used in awide variety of settings, including healthcare, security, and retailstores. These are illustrative examples and do not restrict the scope ofthe present inventive method.

Returning to FIGS. 2A-2F, the system 200 may be embodied as a systemcooperating with computer hardware components and/or ascomputer-implemented methods. The system 200 may include a plurality ofsoftware modules or subsystems. The modules or subsystems, such as thesensor portion 201 and the computer subsystem 298, may be implemented inhardware, software, firmware, or any combination of hardware, software,and firmware, and may or may not reside within a single physical orlogical space. For example, the modules or subsystems referred to inthis document and which may or may not be shown in the drawings, may beremotely located from each other and may be coupled by a communicationnetwork.

The system 270 of FIG. 2F is a high-level hardware block diagram of oneembodiment of the system 200 used to monitor individuals whileprotecting their privacy. The system 200 may be embodied as a systemcooperating with computer hardware components and/or ascomputer-implemented methods. For example, the subsystems, such as thedepth calculation module 210 and all other modules herein, may eachinclude a plurality of software modules or subsystems. The modules orsubsystems may be implemented in hardware, software, firmware, or anycombination of hardware, software, and firmware, and may or may notreside within a single physical or logical space. For example, themodules or subsystems referred to in this document and which may or maynot be shown in the drawings, may be remotely located from each otherand may be coupled by a communication network.

Additionally, the hardware system 200 shown in FIGS. 2A-2F, includingthe various cameras and sensors, in one specific embodiment may beprovided by one or more commercially-available hardware platforms. Forexample, sensor portion 201 may be provided by the Kinect System,available from Microsoft Corporation, or by the Xtion device, availablefrom Asus Corporation. Such commercially available devices may be usedto generate depth data and/or color image data and/or skeleton dataand/or pixel label data. For example, computer subsystem 298 may beprovided by the Xbox System, available from Microsoft Corporation, or bya personal computer, such as one running Microsoft Windows or Apple OSX.

Furthermore, FIG. 2F displays a high-level hardware block diagram of asystem computer 298 that may be used to execute software or logic toimplement the measurements of the user and other steps disclosed in thisdocument. The computer or computer subsystem 298 may be a personalcomputer and may include or connect to various hardware components, suchas the RAM 296, the ROM 297, the data storage 284, and the like. Thecomputer 298 may include any suitable processor or processing device295, such as a subsystem computer, microprocessor, RISC processor(reduced instruction set computer), CISC processor (complex instructionset computer), mainframe computer, work station, single-chip computer,distributed processor, server, controller, micro-controller, discretelogic computer, and the like, as is known in the art.

For example, the processing device 295 may be an Intel Pentium®microprocessor, x86 compatible microprocessor, single core processor,dual-core processor, multi-core processor, or equivalent device, and maybe incorporated into a server, a personal computer, server, remotecomputer, cloud processing platform, or any suitable computing platform.

The RAM 296 and ROM 297 may be incorporated into a memory subsystem,which may further include suitable storage components, such as RAM,EPROM (electrically programmable ROM), flash memory, dynamic memory,static memory, FIFO (first-in, first-out) memory, LIFO (last-in,first-out) memory, circular memory, semiconductor memory, bubble memory,buffer memory, disk memory, optical memory, cache memory, and the like.Any suitable form of memory may be used, whether fixed storage on amagnetic medium, storage in a semiconductor device, or remote storageaccessible through a communication link. A user input 287 may be coupledto the computer 298 and may include various input devices, such asswitches selectable by the system manager and/or a keyboard, or may beconducted independently of such devices, e.g., by using hand gestures orother body gestures, or by using voice commands. The user interface alsomay include suitable display devices 285, such as an LCD display, a CRT,various LED indicators, a printer, and/or a speech output device, as isknown in the art.

To facilitate communication between the computer 298 and externalsources, a communication interface 286 may be operatively coupled to thecomputer system. The communication interface 286 may be, for example, alocal area network, such as an Ethernet network, intranet, Internet, orother suitable network. The communication interface 286 may also beconnected to a public switched telephone network (PSTN) or POTS (plainold telephone system), which may facilitate communication via theInternet. Any suitable commercially-available communication device ornetwork may be used.

The logic, circuitry, and processing described above may be encoded orstored in a machine-readable or computer-readable medium such as acompact disc read only memory (CDROM), magnetic or optical disk, flashmemory, random access memory (RAM) or read only memory (ROM), erasableprogrammable read only memory (EPROM) or other machine-readable mediumas, for examples, instructions for execution by a processor, controller,or other processing device.

The medium may be implemented as any device that contains, stores,communicates, propagates, or transports executable instructions for useby or in connection with an instruction executable system, apparatus, ordevice. Alternatively or additionally, the logic may be implemented asanalog or digital logic using hardware, such as one or more integratedcircuits, or one or more processors executing instructions; or insoftware in an application programming interface (API) or in a DynamicLink Library (DLL), functions available in a shared memory or defined aslocal or remote procedure calls; or as a combination of hardware andsoftware.

In other implementations, the logic may be represented in a signal or apropagated-signal medium. For example, the instructions that implementthe logic of any given program may take the form of an electronic,magnetic, optical, electromagnetic, infrared, or other type of signal.The systems described above may receive such a signal at a communicationinterface, such as an optical fiber interface, antenna, or other analogor digital signal interface, recover the instructions from the signal,store them in a machine-readable memory, and/or execute them with aprocessor.

The systems may include additional or different logic and may beimplemented in many different ways. A processor may be implemented as acontroller, microprocessor, microcontroller, application specificintegrated circuit (ASIC), discrete logic, or a combination of othertypes of circuits or logic. Similarly, memories may be DRAM, SRAM,Flash, or other types of memory. Parameters (e.g., conditions andthresholds) and other data structures may be separately stored andmanaged, may be incorporated into a single memory or database, or may belogically and physically organized in many different ways. Programs andinstructions may be parts of a single program, separate programs, ordistributed across several memories and processors.

Returning to FIG. 8, Step 805 enables the user of the system to store,adjust, and/or transmit measurements or other parameters. These stored,adjusted, and/or transmitted items may be determined by the most recentset of measurements conducted by the system 200; or by any previous setof measurements, conducted by the system 200 in the past, and thenstored for later use; or by any set of measurements provided exogenously(e.g., through ancillary devices or user input); or by any combinationof subsets thereof, including any combination of individual measurementsdrawn across different collections of measurements, includingmeasurements acquired by the system 200 or supplied by the user or byother sources.

The system 200 may interface with, or interact with, an online portalthrough which people may view historic and current measurements and/orschematic outputs, and analytics on those measurements and/or schematicoutputs (for example, graphs or calculations). The portal may be a webbrowser portal, or a portal that is made available through a softwaredownload to a videogame system, or a portal that is made availablethrough an application download to a tablet computer or a mobile phone,or any other type of online interface.

Examples of commercially-available web browsers include MicrosoftInternet Explorer, Mozilla Firefox, Apple Safari, and Google Chrome.Examples of commercially-available videogame systems include MicrosoftXbox, Sony PlayStation 3, and Nintendo Wii. Examples of tablet computersinclude Apple iPad and Samsung Galaxy Tab. Examples of mobile phoneoperating systems include Microsoft Windows Phone, Apple iPhone iOS, andGoogle Android. Embodiments of the present system and method mayincorporate, link to, network with, transmit information to or from, orotherwise employ or utilize any kind of online portal, whether part ofthe system 200 or supplied by a third party, without limitation.

In Step 805, the system 200 may transmit measurements or schematicoutputs or other parameters, for example, to subsystems of system 200(such as data storage device 284), or to external systems or recipients(such as a clinical facility or online database). The measurements orschematic outputs or other parameters may be adjusted in terms offormat, units (e.g. metric vs. imperial), or in any way desired. Themeasurements or schematic outputs or other parameters may be transmittedto, from, or via an online portal, or to, from, or via any other systemor third party. A recipient of measurements or schematic outputs orother parameters may be, for example, a clinician, who evaluates whethera health status decline is likely occurring, and who may choose tointervene, for example, by calling or visiting the patient.

A recipient of measurements or schematic outputs or other parameters mayalso be a caregiver, such as a relative or home aide or friend. Arecipient of measurements or schematic outputs or other parameters mayalso be a social networking system, such as a website or mobileapplication, which may be part of the system 200 or may be provided byor via any other system or third party, and which may utilize themeasurements or schematic outputs or other parameters to share theuser's health status with other individuals in the user's socialnetwork.

Returning back to FIG. 3A, system 300 incorporates aspects of the system200, but augmented with audio sensor 308 and output audio data 309. Forexample, audio sensor 308 may be a microphone or an array ofmicrophones. System 300 is therefore capable of conducting measurementsof the user's voice and speech, in addition to measurements of theuser's body. For example, system 300 may conduct measurements of thevolume and cadence of the user's speech over time. For example, reducedvolume compared to baseline, or slurred speech compared to populationcomparators, may indicate a decline in health status. Measurements ofthe user's voice may also be undertaken to evaluate emotional status—forexample, louder volume than usual, or frequency spectra consistent withangry or depressed tones of voice. References to “measurements”contained herein therefore may therefore refer not just to spatialmeasurements of a user's body, but also audio measurements of the user'svoice and speech.

While various embodiments of the invention have been described, it willbe apparent to those of ordinary skill in the art that many moreembodiments and implementations are possible within the scope of theinvention. Accordingly, the invention is not to be restricted except inlight of the attached claims and their equivalents.

What is claimed is:
 1. A method of monitoring the presence and movementof an individual while protecting the individual's privacy, the methodcomprising: capturing data corresponding to objects positioned within afield-of-view of at least one energy sensor, such that identifying theindividual cannot be ascertained from the captured data, wherein the atleast one energy sensor is configured to capture energy reflected fromthe individual and the objects within the field-of-view; generatingdepth data for the field-of-view based on the captured data from the atleast one energy sensor; identifying elements of the depth data thatcorrespond to the individual within the field-of-view; identifyingelements of the depth data that correspond to an inanimate object withinthe field-of-view; calculating one or more spatial measurements based onthe elements that correspond to the individual and the elements thatcorrespond to the inanimate object, wherein each of the one or morespatial measurements corresponds to a physical dimension; and generatinga schematic output representative of the field-of-view, based on atleast one or more spatial measurements, wherein at least the individualand the at least one inanimate object are labeled in the schematicoutput.
 2. The method of claim 1, wherein the depth data and/or the oneor more spatial measurements are used to generate a schematic output. 3.The method of claim 2, wherein the schematic output is at least onemember selected from the group consisting of: an overhead view, a sideview, a trajectory, a heat map, and a cartoon.
 4. The method of claim 1,wherein the depth data are analyzed for changes in presence or motion,and update-messages or warning-messages are supplied to users,operators, or caregivers.
 5. The method of claim 1, wherein a pluralityof individuals are simultaneously monitored.
 6. The method of claim 5,wherein individuals are identified or distinguished from each other byfacial recognition or by body biometrics.
 7. The method of claim 1,wherein skeleton data and/or pixel label data are generated from thedepth data.
 8. The method of claim 7, wherein the skeleton data and/orpixel label data are used to identify the elements of the depth datathat correspond to the at least one individual.
 9. The method accordingto claim 1, wherein the at least one energy sensor acquires at least oneof reflected infrared light, reflected laser light, reflectedultraviolet light, reflected visible light, reflected X-rays, reflectedmicrowaves, reflected radio waves, reflected sound waves, reflectedultrasound energy, and reflected thermal energy.
 10. The methodaccording to claim 1, wherein the depth data are generated through LIDARmethods.
 11. The method according to claim 1, wherein the depth data aregenerated through pattern deformation methods.
 12. The method accordingto claim 1, wherein at least one additional energy sensor is positionednearby such that the field-of-view of the at least one additional energysensor at least partially coincides with the field-of-view of the atleast one energy sensor, and wherein the at least one energy sensoracquires a first spectrum of energy that overlaps with a second spectrumof energy acquired by the at least one additional energy sensor.
 13. Themethod according to claim 12, wherein the at least one additional energysensor acquires visual light.
 14. The method according to claim 12,wherein the at least one energy sensor and the at least one additionalenergy sensor are co-located in space.
 15. The method of claim 1,wherein a rate of change over time of depth data corresponding to anobject within the field-of-view is used to identify whether the objectwithin the field-of-view corresponds to the at least one individual orthe at least one inanimate object.
 16. The method of claim 1, whereinmorphology of depth data corresponding to an object within thefield-of-view is used to identify whether the object within thefield-of-view corresponds to the at least one individual or the at leastone inanimate object.
 17. A method of monitoring the presence andmovement of an individual while protecting the individual's privacy,comprising: capturing data corresponding to objects positioned within afield-of view of at least one energy sensor, such that identifyingcharacteristics of the individual cannot be ascertained from thecaptured data, wherein the at least one energy sensor is configured tocapture energy reflected from the objects within the field of-view;generating depth data for the field-of-view based on the captured datafrom the at least one energy sensor; identifying elements of the depthdata that correspond to the individual within the field-of-view;identifying elements of the depth data that correspond to an inanimateobject within the field-of-view; calculating one or more spatialmeasurements based on the elements that correspond to the individual andthe elements that correspond to the inanimate object, wherein each ofthe one or more spatial measurements corresponds to a physicaldimension; and generating a schematic output representative of thefield-of-view, based on at least the one or more spatial measurements,wherein at least the individual and the at least one inanimate objectare labeled in the schematic output.
 18. The method of claim 17, whereinthe schematic output of the field-of-view is supplied to an operator, adisplay device, or a storage device.
 19. The method of claim 17, whereinat least the depth data or the spatial measurement corresponding to theat least one inanimate object is retained over time as legacy data. 20.The method of claim 19, wherein the retained legacy data is used toaugment or fill-in the schematic output when an individual in thefield-of-view obstructs a view of the at least one inanimate object. 21.The method of claim 17, wherein the depth data or the one or morespatial measurements are transformed into at least one of atwo-dimensional schematic, a two-dimensional cartoon, a two-dimensionalavatar, a three-dimensional cartoon, a three-dimensional avatar, atwo-dimensional stick figure, and a three-dimensional stick figure.